<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
	<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=iso-8859-1">
	<TITLE>OSBF-Lua</TITLE>
	<META NAME="GENERATOR" CONTENT="OpenOffice.org 2.0  (Linux)">
	<META NAME="CREATED" CONTENT="20050305;1494300">
	<META NAME="CHANGEDBY" CONTENT="Fidelis Assis">
	<META NAME="CHANGED" CONTENT="20060108;21303100">
</HEAD>
<BODY LANG="en-US" BGCOLOR="#ffffff" DIR="LTR">
<CENTER>
	<TABLE BORDER=0 CELLPADDING=2 CELLSPACING=2 STYLE="page-break-before: always">
		<TR>
			<TD></TD>
		</TR>
		<TR>
			<TD>
				<P LANG="pt-BR" ALIGN=CENTER><B><FONT SIZE=4>OSBF-Lua</FONT></B> 
				</P>
			</TD>
		</TR>
		<TR>
			<TD VALIGN=TOP>
				<P LANG="pt-BR" ALIGN=CENTER>Text classification library for the
				<A HREF="http://www.lua.org/">Lua</A> Programming Language 
				</P>
			</TD>
		</TR>
	</TABLE>
</CENTER>
<P LANG="pt-BR" ALIGN=CENTER><FONT SIZE=2><A HREF="#over">overview</A>
&middot; <A HREF="#download">download</A> &middot; <A HREF="manual.html">manual</A>
&middot; <A HREF="#credits">credits</A> &middot; <A HREF="#contact">contact</A>
</FONT>
</P>
<HR>
<H2 LANG="pt-BR">Contents</H2>
<UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><A HREF="#over">Overview</A></P>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><A HREF="#whatsnew">What's
	new</A></P>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><A HREF="#download">Download</A></P>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><A HREF="#installation">Installation</A></P>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><A HREF="manual.html">Manual</A></P>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><A HREF="#credits">Credits</A></P>
	<LI><P LANG="pt-BR"><A HREF="#contact">Contact</A></P>
</UL>
<H2 LANG="pt-BR"><A NAME="over"></A>Overview</H2>
<P LANG="pt-BR">OSBF-Lua (Orthogonal Sparse Bigrams with confidence
Factor) is a C module for text classification written for <A HREF="http://www.lua.org/">Lua</A>.
It is a port of the OSBF classifier implemented in CRM114,
<A HREF="http://crm114.sf.net/"><U>http://crm114.sf.net.</U></A> It
borrows many good ideas from Bill Yerazunis' CRM114, like the
databases basic structure and the Bayesian chain implementation. The
OSBF algorithm is a typical Bayesian classifier but enhanced with the
OSB (Orthogonal Sparse Bigrams) feature extraction technique and an
ad hoc Confidence Factor (aka &ldquo;voodoo&rdquo;), for automatic
reduction of the less significant features impact on the
classification &ndash; noise reduction. The final result is a very
fast and accurate classifier. It was developed focused on 2 classes,
SPAM and NON-SPAM, so the performance for more than 2 classes may not
be the same.</P>
<P LANG="pt-BR">I originally developed both OSB and OSBF for the
CRM114 project, but the OSB technique was officially announced in the
paper &ldquo;<A HREF="http://www.siefkes.net/ie/winnow-spam.pdf"><I>Combining
Winnow with Orthogonal Sparse Bigrams for Incremental Spam
Filtering</I></A>&ldquo;, a work headed and presented by <A HREF="http://www.siefkes.net/">Christian
Siefkes</A> in the 8<SUP>th</SUP> <I>European Conference on
Principles and Practice of Knowledge Discovery in Databases (PKDD)</I>,
on September 2004.</P>
<P LANG="pt-BR">This implementation attempts to put focus on the
classification task itself by using Lua, a powerful language, yet
light-weight and fast, which makes it easier to build and test more
elaborated filters and training methods.</P>
<P LANG="pt-BR">OSBF-Lua is free software and is released under the
GPL version 2. You can get a copy of the license at <A HREF="http://www.fsf.org/licensing/licenses/gpl.txt">GPL</A>.
This distribuition includes a copy of the license in the file
gpl.txt.</P>
<H2 LANG="pt-BR"><A NAME="download"></A>Download</H2>
<UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">The sources can be
	downloaded from <A HREF="http://luaforge.net/projects/osbf-lua">http://luaforge.net/projects/osbf-lua</A></P>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Alessandro Martins
	maintains a package for Slackware at
	<A HREF="http://www.martins.eng.br/slackware/osbf-lua">http://www.martins.eng.br/slackware/osbf-lua</A></P>
</UL>
<H2 LANG="pt-BR"><A NAME="whatsnew"></A>What's new</H2>
<UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[08/Jan/2006] Version
	1.5.3b</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Fixes to the osbf
		module</P>
		<UL>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Fixed the database
			restore function (osbf.restore);</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Changed the osbf.so
			link from absolute to relative to make it simpler to generate the
			Slackware package &ndash; suggested by Alessandro Martins
			&lt;<A HREF="mailto:alessandro@martins.eng.br">alessandro@martins.eng.br</A>&gt;.</P>
		</UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Improvements and
		fixes to spamfilter (v1.1.3):</P>
		<UL>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Better detection of
			the &ldquo;Subject:&rdquo; header line;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Improved scan for a
			command in the subject line. Now it'll detect a command even if
			another filter in the middle mistakenly adds a tag to the
			beginning of the subject line. Problem pointed out by Pavel Kolar.</P>
		</UL>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[01/Jan/2006] Version
	1.5.2b</P>
</UL>
<UL>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Improvements and
		fixes to spamfilter:</P>
		<UL>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">The recover command
			now sends the recovered message as an attachment;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT SIZE=3>Added
			a new config option, <SPAN STYLE="text-decoration: none"><SPAN STYLE="font-style: normal"><FONT FACE="Courier 10 Pitch">osbf.cfg_remove_body_threshold</FONT></SPAN></SPAN>,
			to remov</FONT>e the body of spam messages. Setting
			<FONT FACE="Courier 10 Pitch">osbf.cfg_remove_body_threshold = 20</FONT>
			in spamfilter_config.lua removes the body of all spam messages
			with score greater than 20. The original message is still
			available with the recover command, if needed;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm; font-weight: medium">
			Fixed a problem that occurred when a command-message was sent in
			HTML format. Because of the Content-Type header in the original
			message, the answer, in plain text format, was not visible;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm; font-weight: medium">
			Fixed a bug in the password parsing. An invalid password was
			accepted as OK if it started with the valid password as a
			substring and was the last string in the command.</P>
		</UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Improvements to the
		lib</P>
		<UL>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">New function added,
			<FONT FACE="Courier 10 Pitch">osbf.config</FONT>, to allow
			internal parameter adjustments. This function is more intended for
			experiments and debugging.</P>
		</UL>
	</UL>
</UL>
<UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[15/Nov/2005] Version
	1.5.1b</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Improvements and
		fixes to spamfilter, toer.lua and docs:</P>
		<UL>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">All X-OSBF headers
			were merged into a single one as suggested by Pavel Kolar
			&lt;<A HREF="mailto:kolar@fzu.cz">kolar@fzu.cz</A>&gt;:</P>
			<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><A NAME="line1"></A>Ex:
			<FONT FACE="Courier 10 Pitch">X-OSBF-Lua-Score: 33.63/0.00 [H]
			(v1.5.1b, Spamfilter v1.1)</FONT>;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">White and
			blacklisted messages are now classified too, so that the score in
			the header X-OSBF-Lua-Score is the real one, as if they hadn't
			been listed &ndash; suggested by Pavel Kolar. The subject tags for
			blacklisted and whitelisted messages are the same as configured
			for spam and ham in the config file, respectively;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">The tags in the
			X-OSBF-Lua-Score header don't follow the subject tags defined in
			the config file any more. They're now fixed: [B] [S], [s], [h],
			[H], [W] for blacklisted, spam, spam reinforcement, ham
			reinforcement, ham and whitelisted, according to the
			classification;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">White and black
			lists don't use Lua regex by default any more. There's a new
			option in the config file to turn regex on or off:
			<FONT FACE="Courier 10 Pitch">osbf.cfg_lists_use_regex</FONT>;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Removed the
			trailing spaces from the subject tags in the config file. They're
			now added internally; 
			</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Removed duplicate
			database use info showed by the &ldquo;stats &lt;pwd&gt;&rdquo;
			command;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">The var
			<FONT FACE="Courier 10 Pitch">unlearn_threshold</FONT> in
			spamfilter_commands.lua is now an option in the config file, as it
			should: <FONT FACE="Courier 10 Pitch">osbf.cfg_unlearn_threshold</FONT>;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">More consistent
			thresholds checking in toer.lua;</P>
			<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">DSTTT is now the
			default training method in toer.lua.</P>
		</UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Added the script
		roc.lua, which calculates 1-ROCAC%, a measure of the quality of the
		classifier.</P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[06/Nov/2005] Version
	1.5b - first public release</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Re-tuning of
		internal parameters, after the chain rule fix, resulting in
		improved accuracy.</P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Docs and example
		scripts updated.</P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[30/Sep/2005] Version
	1.4b - internal use only</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Changed
		seen_features and other flags data struture to a separate array of
		unsigned chars, in the learn function.</P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[25/Sep/2005] Version
	1.3b - internal use only</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">C and Lua codes
		updated for lua-5.1-alpha</P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">No more captures in
		string.find</P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Use of new
		string.mach</P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[08/Sep/2005] Version
	1.2b - internal use only</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Fixed an old bug in
		the chain rule that caused bad accuracy with some corpus. It
		sometimes would also cause unexpected worse scores after training,
		as if one had done an &ldquo;unlearn&rdquo;;</P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Fixed a bug in the
		&ldquo;unlearn&rdquo; code that caused broken chains in the
		databases;</P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Implemented a new
		training method acting on both, spam and ham, databases
		simultaneously, doing a &ldquo;learn&rdquo; on the right database
		and an &ldquo;unlearn&rdquo; on the opposite if the score
		improvement was not enough. Now, both toer.lua and spamfilter.lua
		use this new method;</P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[25/Aug/2005] Version
	1.1b - internal use only</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">Changed the training
		method used by the spamfilter. Now the original message is saved
		under a unique SpamFilter ID (SFID) on the server and the original
		message is sent to the user with the SFID added as a comment to its
		&ldquo;Message-ID&rdquo; header. The original message is recovered,
		using the SFID sent back by the user's mail client, in the
		&ldquo;In-Reply-To&rdquo; or &ldquo;References&rdquo; header, when
		he does a &ldquo;Reply&rdquo; for training.</P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[13/May/2005] Version
	1.0b18 - internal use only</P>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">[16/Mar/2005] Version
	1.0b12 - internal use only</P>
	<LI><P LANG="pt-BR">[28/Jan/2005] Version 1.0b1 &ndash; internal use
	only</P>
</UL>
<H2 LANG="pt-BR"><A NAME="installation"></A>Installation</H2>
<P LANG="pt-BR">OSBF-Lua follows the <A HREF="http://www.keplerproject.org/compat">package
proposal</A> for Lua 5.1 and requires Lua 5.1 installed with dynamic
loading enabled. OSBF-Lua was developed and tested under the Lua 5.1
work, alpha and beta versions. It won't work with previous versions.</P>
<P LANG="pt-BR">Installation steps:</P>
<UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">Install
	Lua with dynamic loading enabled:</FONT></P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">For
		linux, execute &ldquo;make linux&rdquo; and &ldquo;make install&rdquo;.
		For other OS, read the instructions in the INSTALL file and your OS
		documentation on how to create shared libs.</FONT></P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">You
		might want to change the occurrences of the O2 flag in CFLAGS to
		O3, in all makefiles, for increased speed.</FONT></P>
	</UL>
</UL>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">Install
	the OSBF-Lua module:</FONT></P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">tar
		xvzf osbf-lua-x.y.z.tar.gz</FONT></P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">cd
		osbf-lua-x.y.z</FONT></P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">&lt;edit
		the &ldquo;config&rdquo; file to suit to your platform &ndash; not
		necessary for Linux&gt;</FONT></P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">make</FONT></P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">make
		install</FONT></P>
	</UL>
</UL>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">You must be root to do the
&ldquo;make install&rdquo; step. If you don't have root access, you
must copy the just created libosbf.so-x.y.z to a directory you have
access to, under the name &ldquo;osbf.so&rdquo;, and add that dir to
LUA_PATH. Ex:</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">mkdir
$HOME/lib</FONT></P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">cp
libosbf.so-x.y.z $HOME/lib/osbf.so</FONT></P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">LUA_PATH=$LUA_PATH:$HOME/lib</FONT></P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">After the osbf module is
properly installed, you may want to install the spamfilter, a Lua
script that uses the OSBF-Lua module to classify and tag messages as
spam or non-spam (ham) according the the score they get, or to the
white/blacklists, if any:</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm; font-weight: medium"><FONT FACE="Courier, monospace">make
install_spamfilter</FONT></P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">The spamfilter files are
installed in /usr/local/osbf-lua. If the dir doesn't exist it'll be
created</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">The next step is to
configure your email account to use the spamfilter:</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">do the following
	steps under your account, not as root</P>
</UL>
<UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">create your local
	osbf-lua dir:</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">mkdir
		$HOME/osbf-lua</FONT></P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">create your log dir:</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">mkdir
		$HOME/osbf-lua/log</FONT></P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">copy the spamfilter
	config file to your dir:</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">cp
		/usr/local/osbf-lua/spamfilter_config.lua $HOME/osbf-lua</FONT></P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">edit
	spamfilter_config.lua to set your password</P>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">change the current
	dir to your osbf-lua dir and create the spamfilter databases</P>
	<UL>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">cd
		$HOME/osbf-lua</FONT></P>
		<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">lua
		/usr/local/osbf-lua/create_databases.lua</FONT></P>
	</UL>
	<LI><P LANG="pt-BR" STYLE="margin-bottom: 0cm">add the following
	lines to your .procmailrc 
	</P>
</UL>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">#
set OSBF_LUA_DIR to where spamfilter.lua, spamfilter_command.lua etc
were installed</FONT></P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">OSBF_LUA_DIR=/usr/local/osbf-lua</FONT></P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">OSBF_LUA_USER_DIR=$HOME/osbf-lua</FONT></P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">#
let the Lua interpreter find the &ldquo;osbf&rdquo; module. </FONT>
</P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">#
uncomment if you installed a local copy of the osbf module (e.g. no
root access)</FONT></P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">#LUA_PATH=$LUA_PATH:$HOME/lib</FONT></P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">:0fw:
.msgid.lock</FONT></P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">*
&lt; 350000 # don't check messages greater than 350000 bytes</FONT></P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><FONT FACE="Courier, monospace">|
$OSBF_LUA_DIR/spamfilter.lua --gdir $OSBF_LUA_DIR --udir
$OSBF_LUA_USER_DIR</FONT></P>
<P LANG="pt-BR" STYLE="margin-left: 2.01cm; margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">Check your installation
sending a message to yourself with the following command in the
subject line:</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">help
&lt;your password&gt;</FONT></P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">You should receive a
message with a help on the spamfilter. Then, send another command in
the subject line to verify that the databases were created correctly:</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">stats
&lt;your password&gt;</FONT></P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">You should get a
statistics report on the just created databases.</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">From now on, all messages
you receive with less than max size specified in the procmail recipe
will be classified and tagged according to the score they get:</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<TABLE WIDTH=707 BORDER=1 CELLPADDING=4 CELLSPACING=3 STYLE="page-break-inside: avoid">
	<COL WIDTH=88>
	<COL WIDTH=592>
	<THEAD>
		<TR VALIGN=TOP>
			<TD WIDTH=88>
				<P LANG="pt-BR" ALIGN=CENTER><FONT FACE="Courier, monospace">Tag</FONT></P>
			</TD>
			<TD WIDTH=592>
				<P LANG="pt-BR" ALIGN=CENTER><FONT FACE="Courier, monospace">Meaning</FONT></P>
			</TD>
		</TR>
	</THEAD>
	<TBODY>
		<TR VALIGN=TOP>
			<TD WIDTH=88>
				<P LANG="pt-BR" ALIGN=CENTER><FONT FACE="Courier, monospace">[--]</FONT></P>
			</TD>
			<TD WIDTH=592>
				<P LANG="pt-BR"><FONT FACE="Courier, monospace">almost sure it's
				a spam - score &lt;= -20</FONT></P>
			</TD>
		</TR>
		<TR VALIGN=TOP>
			<TD WIDTH=88>
				<P LANG="pt-BR" ALIGN=CENTER><FONT FACE="Courier, monospace">[-]</FONT></P>
			</TD>
			<TD WIDTH=592>
				<P LANG="pt-BR"><FONT FACE="Courier, monospace">probably it's a
				spam (reinforcement zone) - score &lt; 0 and &gt; -20</FONT></P>
			</TD>
		</TR>
		<TR VALIGN=TOP>
			<TD WIDTH=88>
				<P LANG="pt-BR" ALIGN=CENTER><FONT FACE="Courier, monospace">[+]</FONT></P>
			</TD>
			<TD WIDTH=592>
				<P LANG="pt-BR"><FONT FACE="Courier, monospace">probably it's not
				spam (reinforcement zone) &ndash; score &gt;=0 and &lt; 20</FONT></P>
			</TD>
		</TR>
		<TR VALIGN=TOP>
			<TD WIDTH=88>
				<P LANG="pt-BR" ALIGN=CENTER><FONT FACE="Courier, monospace">[++]</FONT></P>
				<P LANG="pt-BR" ALIGN=CENTER><BR>
				</P>
			</TD>
			<TD WIDTH=592>
				<P LANG="pt-BR"><FONT FACE="Courier, monospace">almost sure it's
				not spam &ndash; score &gt;= 20. This tag is here just for
				symmetry, it's not used. An empty tag is used in place of it so
				as not to pollute the messages.</FONT></P>
			</TD>
		</TR>
	</TBODY>
</TABLE>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">If the classification is
wrong you must train the filter replying (you must do a &ldquo;Reply&rdquo;,
not a &ldquo;Forward&rdquo;) the message back to yourself, replacing
the subject with the correspondent training command:</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">learn
&lt;password&gt; spam </FONT>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">or</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">learn
&lt;password&gt; nonspam</FONT> 
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">The body of the message
may be erased, it's not required. The original message, temporarily
saved on the server by the spamfilter, will be recovered through the
SFID (Spam Filter ID), a special mark added to the header when the
message arrived to the server.</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">If you make a mistake, you
should undo the training with the command &ldquo;unlearn&rdquo;. Ex:</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Courier, monospace">unlearn
&lt;password&gt; spam</FONT></P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><FONT FACE="Bitstream Vera Serif">if
wrongly trained as spam.</FONT></P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm"><BR>
</P>
<P LANG="pt-BR" STYLE="margin-bottom: 0cm">Training when the
classification is wrong is essential for accuracy. Training when in
the reinforcement zone, called <I>reinforcement</I>, is highly
recommended for increasing and keeping the accuracy high. After you
have a well trained filter, say 99% or better accuracy, you may want
to reduce the reinforcement zone, eg. [-10, 10], so as not to have to
do many reinforcements a day. You may change the reinforcement zone,
tags, etc, by editing the spamfilter_config.lua file.</P>
<H2 LANG="pt-BR"><A NAME="credits"></A>Credits</H2>
<P LANG="pt-BR">The OSBF-Lua lib and spamfilter.lua were designed and
implemented by Fidelis Assis, who holds the copyright. 
</P>
<P LANG="pt-BR">The OSB technique, as well as the OSBF classify and
learn codes are based on the OSB and OSBF I originally developed for
the CRM114 project (<A HREF="http://crm114.sf.net/">http://crm114.sf.net</A>),
as a derivative work based on Bill Yerazunis' CRM114 Markovian
classifier.</P>
<H2 LANG="pt-BR"><A NAME="contact"></A>Contact</H2>
<P LANG="pt-BR">For more information please <A HREF="mailto:fidelis-NO-SPAM-THANKS@pobox.com">email
me</A>. Comments are welcome! 
</P>
<HR>
</BODY>
</HTML>